US005682326A 


United States Patent [19] 

Klingler et al. 


[U] Patent Number: 5,682,326 

[45] Date of Patent: Oct. 28, 1997 


[54] DESKTOP DIGITAL VIDEO PROCESSING 
SYSTEM 

[75] Inventors: Joseph W. KHngler; Clifton L. 

Vaughan, both of Toledo, Ohio; 
Gregory M. Millar, Pleasanton. Calif. 

[73] Assignee: Radius Inc., Sunnyvale, Calif. 

[21] AppL No.: 415354 

[22] Filed: Apr. 3, 1995 

Related US. Application Data 

[631 Continuation-in-part of Ser No. 924,714, Aug. 3, 1992, Pat 
No. 5,404,316. 


Animati on Production Environment (apE) Version 2.0 
User’s Manual, copyright 1990. Ohio State University, title 
page and pp. 8-9. 

D. Ingalls et al. “Fabrik: a visual programming environ- 
ment,” in OOPSLA *88 Conf. Proc., pp. 176-190 (1988). 
RE. Hacberli, “ConMan: a visual programming language for 
interactive graphics,” SIGGRAPH *88 Conf, Proc., Coraput 
Graph. 22(4) pp. 103-111 (1988). 

MP. Stovsky et aL, “Building interprocess communication 
models using yiTLE," in Proc. Hawaii Int, Conf. Systems 
Sciences, vol. 2, 639-647. IEEE (1988). 

Primary Examiner — Alpus H. Hsu 

Assistant Examiner — Seema S- Rao 

Attorney, Agent, or Firm — Brooks & Kushman P.C. 

[57] ABSTRACT 


[51] Int a 6 G06K 15/90 

[52] U.S.a 364/514 A 

[58] Field of Search 364/514 A; 348/563. 

348/564. 578. 581. 584. 590. 588; 358/310. 
311; 395/100. 161. 114. 116-118. 153-157. 

159 

[56] References Cited 


U.S. PATENT DOCUMENTS 


3,617.626 

11/1971 

Bluth et al 

358/311 

4,538,188 

8/1985 

Barker 

360/143 

4,635,136 

1/1987 

Ciampa et al. .... 

364/948.11 

4,660,101 

4/1987 

Martin - 

358/342 

4,675,755 

6/1987 

Baumdster et al. 

.. 348/26 

4,685,003 

8/1987 

Westland 

. 358/335 

4,717,971 

1/1988 

Sawyer ............... 

358/311 

4,729,044 

3/1988 

Kiosel 

360/14.1 

4,873,585 

10/1989 

Blanton et al 

... 358/335 


(List continued on next page.) 

OTHER PUBLICATIONS 

Abode Premier product circular, printed Oct 1991. 

DiVa VidcoShop product circular. 

Digital Equipment Corp. Advanced Visualization System 
(AVS) User’s Guide, May, 1992, cover page, and pp. 1-5, 
1-6, 1-7, 1-8. 1-9, 1-10, 1-11 and 1-12. 


A digital image processing system includes a 
microprocessor, random access memory, storage memory, 
input display, and suitable logic to prpyide ^ting^digital 
video image.proces sing operations in ^graj^cai^us^interr 
face which allows the user to quickly'and effidently edit a 
niow;'The system includes logic for display ing^the, 
^assembled video and audio media, transitions, and special 

^ wHiS^SK^^Be^^te^operatcar^^^jinear >rle.w of the 
fno^e*"S®^elFas a qihck understanding, of. the dataflow 
( ass^3^ t ^tIf rf th*e Ajr p¥o^si^^f 1f Sie ccMistituents of the 

m^^S^S^»ihfin"tim^ In one embodiment, the system 

presei^es tfe^dataflow ^associated with processing & video V 
chgby storing the information associated with that dip ma' 

^media in a conventional format but also indudes an addi- 
tional data seg ment h aving the a ppropriate indices toany j 
source clips and^imflge rt .pn^ssing„eff<xts/ij]clLiding the 
necessary timing and parameters for reproduction^ ofjhe 
resultant chp?The sy stem alsopreferably includes logic for 
creating, mnintai ning and displaying a separate render track 
which maintains the media corresponding to the portions of 
the program that have already been rendered in proper 
synchronization with those portions of the program that have 
been edited since the last rendering, thereby limiting any 
future rendering to those portions of the program which 
require reprocessing since the last rendering of the program 
of the movie. 


4 Claims, 9 Drawing Sheets 
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DESKTOP DIGITAL VIDEO PROCESSING 
SYSTEM 

related application 

This application is a continuation-in-part of application 
Ser. No. 924,714, U.S. Pat No. 5,404316, filed Aug. 3, 
1992. 

BACKGROUND ART 

Ed iting videotape using conventional equipment and 
methods can be expensive and tedious. The process is 
essentially linear (i.e., modification of an earlier portion of 
a movie affects, and requires reprocessing, of the later 
portions of that movie), resulting in a restrictive editing 
environment which indudes significant reprocessing of 
unedited media. 

A number of digital video image processing and editing 
systems have been developed for use on desktop computer 
systems. Existing desktop digital video editing systems 
typically provide the capability for arranging digital movie 
dips in a desired sequence, and processing selected portions 20 
of these clips to, for example, provide transitions between 
clips or special effects. In order to produce a dig it al movie. 
These existing systems also typically indude the capability 
of writing the finished digitized movie to another media, 
such as videotape. These systems may also employ various 25 
compression techniques to reduce the amount of digital data 
which defines a particular movie. However, even in a 
compressed format, significant processing time is required 
to implement transitions and special effects in the digital 


purpose desktop computer to q uickly andearily edit movi es 
i id' a non-linear environment 

object of the present inventio n to provide a 
vidco^digital ini^e jrf6<^slhg system including an interfaced 
5 jwtoch allows the usef tb assdnble and^edit a movie in_an 

which allows for 


^whidi a ' 

< are^ion, organization , 4nd 

quicFrecoveiy of movieci^arrangement and assemblyjrf 

desired 

10 tian ritio^^ effeds, and expeditious rendering^of 

^e Kmsbed movie'T 

Itisan other object of the present invention to provide a 
digital image processing system requiring minima] process- 
ing to render the digital media corresponding to a completed 
movie. 

It is anotira .objert of Jhe present invention Jo provide a 
Satasttua ure for be 

stored and recovered to both play back the digital movie, and 


is 


stored media - from the source media- from which it was 
created. 

It is yet another object of the present invention to provide 
a digital video processing system which automatically 
records the processing operations selected to perform a 
particular video edit, so that variations in the editing process 
can be accomplished by recalling the original editing opera- 
tions and reprocessing only those portions of the editing 
operations necessary to achieve a different desired result. 

regime Thus, the processing, and reprocessing, of the digital » 10 «* the «bove ®nd other objects, the digital 

video image clioT during editing «urbe tiiL-coosundng. bnage processing system of die present invention includes a 
viaeo image cups ounng cuwug « & processor, random access memory, storage memory, such as 

magnetic disk, input means, display means, and suitable 

i * i I Al »'tn1 np/vwclnn nrvpro frnne 


video image clips during editing can be time-consuming, 
particularly where the entire movie is reprocessed each time 
a portion of it has been edited. And, while existing di git al 
video systems provide the video editor with a non-linear 
editing environment, the user interface of the systems is 
often cumbersome. 

Also, in existing systems, once the arrangement, cutting, 
and editing of a movie has been completed, the digital media 
corresponding to all of the frames of the movie must be 


33 


logic to provide existing digital image processing operations 
in a graphical user interface which allows the user to edit a 
movie in the less restrictive non-linear digital image envi- 
ronment including multiple integrated windows which allow 
for quick and easy creation, organization, and assembly of 
clips into a movie. 

The system of the present inven tion includes lo gic for 

« l .1 1 - - -r- J « Ir .n eiti/vnfl 


processed, ot rendered. When the rendered media is 40 
generated, a complete digital video Is created which can be 


and s pecia l effects/ina scries erf independent Jracks,_each 
correlated to a third parameter, suc h as a hori zontal time linen 
The sequence of the clips, and thcTiocation/and duration or 
transitions and special effects, as well as the associated audio 


generated, a complete digital video 

played back on the appropriate available hardware. 

However, all of the information required to assemble the 
movie, such as the identification and location of the portions 
of the source dips utilized, the transitions and spedal 45 transmons ana special areas. as wcu a, «ie 
effects, and the associated processing and timing inform*- tracks, can be correlated at a glance to a honzontal tune line 
tion is lost, or stared in another unrelated project file. 

The words ‘Video” or “movie” as used herein are intended 
to refer to any electronic display erf digital data sequences as 
images (and, optionally, sound). As used herein, video (ot 50 
movie) media may include one or more “clips” each erf 


which are subsets of the movie. Each clip is itself a movie 
comprising a sequence of one eff more frames of images 
(and, optionally, sound). As used herein, a “program” is a 
meanin gful sequence of clips, transitions, and special 
effects, assembled by an editor into a movie. 

It should be noted that the words “desktop” or “personal” 
computer as used herein are intended to refer to any of the 
currently available general purpose single-user computers 
which may operate independently or network and may 
incorporate suitable software to perform other, non-video 
processing functions, such as word processing, graphics, 
database, and spreadsheet functions. 

DISCLOSURE O F INVEN TION — 

It is, th erefore, an object of the present inve ntion,to creat &^ 
a digital image proces sing sy st em whirfTutilizes a general ^ 


'65 


a 


0 


of the movie. Moreover, the vertical arrangement of the 
display of the independent trades allows for quick under- 
standing of tiie dataflow associated with the processing of 
the constituents of the movie at any point in time. 

The system alsojncludcs Jogic for maintaining the media 
and [special effects ( M EX?) tracks as independent data 
entitiesTwhidf are synchronized (or correlated) to a common 
reference such as SMPTE codes, or other references to a 
55 common time-lin e car fr ame n umber. I t will bejip preci ated 
that the separation of the media clips from effects provides 
an additionaFrcntiolloW^ incl u din g the abifity/t o 

replace media without affecting the effect its elf. the abilityto 
controlythc.pacingi w CT gl jtoe )[of an effect independent of 
60 ^p le and thejil^ty to save an effec t and 

'reapply it Jo a nothe r time,, segment 

The system of the present invention may also indude 
suitable logic for creating dataflow graphs recording the 
image processing operations, and associated parameters and 
inputs selected using the interface, thereby allowing for easy 
modification of a previously designed set of processing 
operations and reprocessing of one or more selected video 
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movie clips through recollection* alteration, and reprocess- 
ing of the stored dataflow graphs relating to the selected 
dips. In one embodiment* the present invention effectively 
preserves the dataflow associated with processing a selected . 
dip by storing the information associated with that dip in a 3 
comp osite data fil ejwhicfa indudes not only the digital movie 
media in a conventional format but also indudes an addi- 
tional data segment having the appropriate indices to any 
source clips and Image processing effects, including the 
timing and parameters which produced the resultant clip, as 
w ell as s uitable synchronization information for recreating 
flie^resuMn^tl$f 

The video image processing systemcfjhe present.systera 
also includes an interface capable of (jjj^ayin g one or more 
mi grated mo vie-editing windows. , 

sSB989Sij^seI^^fd^^ | S^^£ffipi^ss,.playmg 1 
in real tune along side a display of th e.source dips. Another 
window displays reference dips! either in a list or formatte d 
as a two-dimensional array of clips with graphi cal identifi er? 
which may be arranged in the array in a desired sequence for 
assembly into^a moyie. A third window is formatted ? to 
display the sclected cli^ txansitions, and special effects as 

video an d audi cT media, as well as any image processing 
effects associated with the media , in a time line view to 
provide more precise display of those clips for frame- 
accurate editing. These windows, known as the Monitors, 
Project, and Sequencer Windows, res pectivel y, jog ether pro- 
vide a complete composition and editing environment which 
allow the user t o organize an d process movie cli ps into a 
desired end product^ : 30 

The Monitors Window is, in substantial part, two frame- 
sized sub-win dows. One of the fra me- sized su b-windows is 
aToiaccToo^ot whi ctTdisplay s the di gitized so urce dips', 
TOT adj aoenTTul^iSciow affowslhe user to view one or 35 
iWe selected-dips that have been deflnerfantfftM- processed 
using the editing system of the present invention. A Monitors 
Window'includes standafd^controls to'start and stop the 
movies in either of the sub-windows, as well as control the 
sound volume, and define in and out points to create ^ 
reference clips. 

In one embodiment of the invention, a second, external 
video mo nitor is utilized, preferably to dis play t he Monitor s 
Window, clips and selected reference clips 

can be viewed In the Monitors Wind^ 45 

is simult^e and w orkin g in other, system^ 

editing windows chsplayed on the primary^omputer^tohi-; > 

The Project Window preferably provides two alternate 
views of the reference dips collected for a particular movie 50 
project In the Text View, the Project Window displays an 
alpha-numeric list of selected dips. In the Picture View, the 
Project Window displays some or all of the selected dips on 
an adjustable mxn array wherein each of the clips is iden- 
tified by a selected image frame and, optionally a title bat 55 

The Sequencer Window provides a graphical time-line 
view of the selected video and audio media along with the 
associated processing effects applied to the media to yield 
the resultant movie. The Sequencer Window is particularly 
useful for assembling selected clips into a movie, since file 60 
linear view of the video and audio material and the display 
of the synchronization of the effects applied to that media, 
facilitate quick, detailed editing decisions. The output of any 
editing operations are synchronized with, and may be 
viewed on the Monitors Window. 63 

The present invention also preferably includes logic for 
correlating each media track, such as for example the first 


video track, with any/FX tracks associated with that media 
track. In one embodiment ~ of the invention, the system 
correlates associated media and FX tracks with a common 
time-line in SMITE time code units. It will thus be appre- 
ciated that not only does the system of the present invention 
provide a graphical correlation of each media track and its 
associated effects with run time (in the Sequencer Window), 
it also organizes the data associated with these entities in 
independent but time-correlated files. Thus, a particular 
effect may be initiated during a selected time period (Le., 
inserted in the appropriate FX track at selected SMFTE in 
and out points) independently of the affected video media. 
Conversely, a clip can be removed from the video track, and 
another inserted, without changing the effect applied during 
that time period 

The system also preferably includes logic for creating, 
maintaining and displaying a separate render track which 
m ai nt ains the media corresponding to the portion(s) of the 
program that have already been rendered in proper synchro- 
nization with those portions of the program that have been 
edited since the last rendering. By recording any edits to file 
program and determining the effect of those edits on any 
downstream (i.e., later-in-time) portions of file program, the 
render track automatically keeps track of those portions of 
the rendered program which must be re-rendered at any 
point in time. When the program is rendered, the system 
processes the source media referenced in all of the reference 
clips (i.e., all of the clips in the Sequencer Window tracks) 
utilizing the selected instructions for adding transitions and 
filters. Once rendered, a single media unit — a video 
program — is created for review or, if desired, for printing to 
videotape. The processing time associated with the applica- 
tion of special effects filters and transitions is significant 
The render track, however, shortens this processing time, 
and thus speeds up the rendering of the movie, by 
re-rendering only those clips which have been edited, or 
which are affected by dips which have been edited, since file 
last rendering. 

The present invention also preferably accomplishes many 
of the image processing techniques on multiple key frames 
of selected clips by progressively altering the frames of the 
designated clip on the basis of modified parameters inter- 
polated (or extrapolated) from specified parameters for the 
key frames. 

The present invention also provides a toolbox of digital 
image processing filters for movies which can be defined, 
combined, and previewed in a visual programming environ- 
ment 

These and other objects, features, and advantages of the 
present invention are readily apparent from the following 
detailed description of the best mode for carrying out the 
Invention, taken in connection with the accompanying draw- 
ings. 

BRIEF DESCRIPTION OF DRAWINGS 

FIG. 1 is a schematic view of the system of the present 
invention; 

FIG. 2 illustrates a Digitizer Window; 

FIG. 3 illustrates a Monitors Window; 

FIG. 4 illustrates a Project Window in Text View; 

FIG. 5 illustrates a Sequencer Window; 

FIG. € illustrates a dataflow chart which may be derived 
from seeing the media and FX tracks in the Sequencer 
Window; 

FIG. 7 illustrates a Sequencer Window with an expanded 
view of a transition between two dips; 
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FIG. 8 is the dataflow diagram of a representative tran- 
sition and a representative effect performed on selected 
frames of two adjoining dips; 

FIG. 9 is a flowchart illustrating the rendering logic 
employed in one embodiment of the invention; 

FIG. 10 illustrates the Monitors Window in FX View; 

FIG. 11 illustrates fee Monitors Window in Trim View; 

FIG. 12 illustrates the Project Window in Picture View; 
and 

FIG. 13 is a schematic representation of a composite clip 
utilized by the system of the present invention. 

BEST MODE FOR CARRYING OUT THE 
INVENTION 

Referring now to FIG. 1, the video image processing 
system of the present invention generally Indicated as 20 
indudes a conventional desktop computer 22 including at 
least one processor 24, random access memory 26, storage 
memory 28. and suitable logic 30 to implement the image 
processing operations in the visual programming environ- 
ment as hereinafter described. The system also indudes at 
least one display 32, and one or more input devices such as 
a conventional keyboard 34 and/or mouse 36. The system 


user interface including^ conventional menu bar from 
which any of a r numb er of window/ may be opened and 
utilized in editing a digital movie. As with other conven- 
tional Graphical User Interfaces (GUTs). the commands 
available under each of the menus may be displayed by 
keystroke command, or by positioning the arrow over the 
desired menu name and clicking to “pull down" the avail- 
able commands. Again, as with conventional pull down 
menu interfaces, the commands are invoked by holding the 
mouse button down and positioning the arrow 66 down the 
menu until the desired command is highlighted, then releas- 
ing the mouse button. A desired command may also be 
invoked by striking the key combinations listed in the 
pull-down menu opposite the command. Consistent with 
conventional GUI's, the user may move from an active 
window to another window by positioning the cursor to the 
other window and “clicking" on it. Alternatively, the user 
may position the cursor on an icon associated with the 
desired window or action and “click" on the icon. Again, as 
with conventional GUI's, die user may double click on a file 
to load the data in the hie or, if the hie is aprogram. run the 
prog ram For example, die user may position the cursor over 
the clip displayed in fee Monitors Window and double click 
to play feat clip. 


also preferably includes a second display 38, such as an ^ n should be noted feat, in the embodiment of the inven- 
extemal television standard (e.g., NTSC) monitor. tion w hich employs a second video monitor 38, each of fee 

In one embodiment, the system utilizes an Apple Madn- monitors 32 and 38 in the system are configured for a 

tosh series, having a Motorola 680X0 (68020, 68030, and seamless mouse movement. For example, if fee external 

68040) series microprocessor, at least 32 megabytes of monit^38Ts i sitting'to w the left of the computer monitcr 32 

RAM, at least 200 megabytes of disk memory, a conven- when in use, the system is configured so that, to move the 

tional keyboard with numeric keypad, and conventional mouse pointer from a point in fee display of fee computer 

mouse cr roller ball. This embodiment of fee system also monitor 32 to a position on the display erf fee extmial^ 

employs Apple System 7.1 (or later) operating system, monitor 38, the operator may drag fee mouse off the left side 

QuickTime 2.0. Macintosh Drag and Drop Manager, Apple of thc computer monitor screen to see it appear on fee right 

Sound Manager 3.0 (or later), and QuickDraw GX 1.0 (or 35 .edge of the external monitor. Again, this seamless integra- 
later) video software. tion of fee mouse pointer wife multiple monitors is achieved 

Other commercially available general purpose desktop through the use of conventional monitor control software, 
computer hardware with similar color graphics capabilities FIG. 2 illustrates a Digitizer Window which is used in one 
and comparable digital video support software may also be embodiment of fee present invention. The Digitizer Window 

utilized for this purpose. 40 may be utilized to control an external video machine to 

Logic suitable for providing the image operations, file capture and digitize the source video, . as well as_selectinand 

utilities, and the visual programming interface hereinafter outpcuntstodefinefee sourcccl^s.^eclipsj^tnefeinbly 
described are preferably programmed in a Unix language defin^l^a*^ out — $*££)&/ 

such as “C" or X++." Again, however, other similar general (prefc^ 

purpose programming languages may be utilized to imple- 45 .relatihg^tb feat^cUp^Q ^C g^^^y^^^ur^clips^either 
ment fee logic and interface described herein. ^infeviduaHyTor ijt groi^^clfeep fegirized. 

ft will be appreciated that the system of the present ^ft'wflTbe appreciated feat, while the system 20 may 
invention can be implemented wife features found in other provide a Digitizer Window or other external device control 
image processing software systems, such as those illustrated window as described above, other conventional techniques 
and described in U.S. Pat. No. 5,404 3 16. which is hereby 50 a nd eq uipment may be utilized to digitize and mark' the 
incorporated by reference for that purpose. In particular, fee sourer dip(s\ which may then be edited as described herein 
Sequencer Window with its independent, time-correlated onthesystem 20. 

media tracks, FX tracks, and rendering track, as well as fee Referring to FIG. 3, after digitizing, the clips can be 
composite clip data file structure can be implemented in a reviewed in fee Monitors Window 46. In particular, the 
variety of video image processing environments, such as 53 digitized source clips are displayed in the source monitor 
that disclosed in U.S. Pat. No. 5.404316, without departing sub-window 48 of fee Monitors Window 46. The Monitors 


from fee spirit of the present invention. 

It will also be appreciated that, in particular, the dataflow 
concepts, as described in FIGS. 2, 16 and 17, and the 
attendant specification of U.S. Pat. No. 5,404316, may be 
incorporated into fee system of the present invention, to 
provide an impro ved digital video movie editor. Again, the 
disclosure of U.S. Pat No. 5,4043 16 is also incorporated by 
reference herein to the extent feat it pertains to these 
dataflow concepts. 

Referring now to FIGS. 1-5, the display 32 (or, 
optionally, the displays 32 and 38) provide(s) a graphical 


Window 46 also includes graphical button controls 50 
corresponding to conventional video players. Reference 
clips can be created from source clips by viewing selected 
60 portions of the source clips in the source monitor sub- 
window 48, selecting in and out points, and namin g each of 
the reference clips. A slider-bar or^ other suitable^conffoLs 
allow fee user to . define be ginning and end points/for 
reference clips. Thus, playing fee source clips in fee Source 
65 Monitors sub-window 48 allows the user to view exactly 
what has been digitized to disk (the sour ce clips) as well as 
fjpfinr reference dips to be utilized in assembling a program. 
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As with the other windows comprising the Interface of the 
present invention, suitable conventional graphical buttons, 
slider bars, or other graphical switches or controls may be 
included in the windows to effect “point and dick” activa- 
tion of the functions within the windows. Alternatively, or in 
addition, pull-down menus, and/or selected keyboard com- 
mands may be implemented to provide the user with he 
ability to input information and/or control the various func- 
tions performed by the system. 

Once the reference clips have been defined and collected, 
a program can be cr eated usin g the editin g system 20 of the 
p resent inven tion, ^pro gram is a m aeaningfiil sequence of 
reference clip?* createdfromlhe sotmcechpS. A reference 
dipMoesnot'contain the actual digitized media. Instead it 
contains references back to the media in the source dips, 
including in and out points (clip beginning frame and dip 
ending frame), for that reference clip. It should be noted that 
the reference clips defined and created for a particular 
program do not necessarily have the same in and out points 
as he source clips which contain the media referenced in he 
reference clips. 

Once created, the reference clips for a particular movie 
project may be organized in the Project Window 60. As 
shown in FIG. 4, the Project Window may be displayed in 
jTtext V iew, resulting in he display of all of he clips^in a 
particular project group or bin. In Text View thcProject 
Window 60 lists certain selected information about all of he 
dips in a particular bin in he r dip- sub- wi ndow 6 2. This 
infommt ioD typically includes the clip name, dip duration, 

codes^TtiMT 

nu m b er and of single cUp. 'comm^ts, 
status and ta r^ ID.* The status of.a dip~t^icaUy'indirctes 
wJghCTh^cl^is on 

9* t0 radia hat has^Mn^o^ttized and stored, on 

CttSt'TL- L 


8 


to 


15 


25 


f&sk. The dips" aregroupedf in bins which are separatdy 
L displayed in he bin sub-window 64 of he Project Window. 

Referring now to FIG. 5, he system 20 of he present 
invention includes a Sequencer Window 70 which provides 
a graphical representation of a time line 72 and multiple 
associated tracks 74-83, into which reference dips contain- 
ing video and/or audio media, and processing functions, 
such as clip transitions 84 or special effects filters 86, can be 
arranged and correlated in time. A program, that is, a 
collection of reference clips, may hen be assembled in the 
Sequencer Window 70 by opening the desired number of 
video, FX. and/or audio tracks, and arranging he media 
and/or effects in he desired sequence. As illustrated by he 
example in FIG. 3, a program can, far example, be created 
to indude some potion or all of the ‘ledge shot” video dip 
90 followed by he “be on glacier” clip 92 and thereafter 
followed by he “twist shot” clip 94. These clips can be 
dragged into he Sequencer Window 70, either from he 
Project Window 60 or from he Source Monitor sub-window 
48 of the Monitors Window 46 by using conventional drag 
and drop techniques. The starting point and ending point of 
he reference clips can be changed, and/or the clips can be 
moved relative to other clips simply by clicking on and 
dragging he graphical representation of he clip in he 
sequencer time line. 

The amount of detail seen in he time line window may 
also be varied by activating the zoom in and zoom out 
buttons 96 and 98 on the window. A Sky View subwindow 
100 may also be provided. The Sky View subsubwindow 
illustrates he entire program, preferably as color-coded 
tracks, and includes an indicator 102 which identifies the 
portion of the program that is currently visible in the 
Sequencer Window 70. 


As he program is graphically assembled in he Sequencer 
Window 70, some portion or all of he program may be 
viewed in the program monitor subwindow 54 of the 
Monitors Window 46. The portion of the prog ram to be 
viewed in he program monitor subwindow 54 may be 
selected by moving he time line cursor 106 over he desired 
starting frame of the dip. Alternatively, or additionally, he 
program monitor subwindow 54 may provide access to 
select any of the clips in he program assembled in the 
Sequencer Window 70 via a conventional pop-up menu 
which can be activated by pressing a program menu button 
(net shown) or other suitable keystroke command. When a 
clip is selected from he pop-up menu in he program 
monitor subwindow 54, he time line cursor 106 in the 
Sequencer Window 70 moves to he beginning of that clip. 
If the clip is not currently visible in he Sequencer Window 
70, the view in the window changes to display he clip at he 
center of the window. 

^E^^™tg agai n to FIG. 5, he s ystem preferably provides 
20 a ^plurali ty- of-[ ro^defijted special ~effects~which~ may be 
applied'to he selected dips as desired. In'one^mbodimcnt, 
toese effects are applied using conventional filters, including 
color adjust, fade, mirror, picture-in-picture, tint and titling. 
The desired filter is selected from a pop-up menu which is 
accessed using conventional menu activation commands, 
and the filter is selected and dragged into an FX track 86 in 
the Sequencer Window. The parameters for controlling he 
operation of he filters may be set for selected key frames via 
^ a filter control window which, again, is activated by using 
^30 conventional menu activation techniques. 

The system 20 preferably allows he user to specify filter 
parameters for two or more key frames. The system then 
calculates he appropriate progressive values to allow he 
filter to gradually change he value set in one key frame to 
the values set in he next adjacent key frame so that he 
filters affect on all cf the frames between two adjacent key 
frames may be interpolated. It will be appreciated that by 
using multiple key frames, the system allows he user to 
change he rate of change effected by the filter over the 
40 length of the filter. It will also be appreciated that, while in 
one embodiment of the present invention the interpolation 
between key frames is linear, other non-linear, and/cr dis- 
continuous rates of interpolation can be implemented, using 
mathematical models or other data sources as modulators, to 
45 allow he user the capability cf dynamically changing the 
effect of a filter without specifying individual filter param- 
eters for each frame of the media affected by he filter. For 
example, a known “spline” (or smooth curve) may be 
utilized as a mathematical template for interpolation 
so between key frames. Also, in place of interpolation, the 
system may provide a “hold” feature which identically 
modifies all of the frames subsequent to a key frame in an 
identical manner to that defined for that key frame, then 
abruptly changes he effect at he next key frame (and 
55 hereafter if appropriate) according to he parameter set for 
that key frame. Other templates may be u tilize d for inter- 
polation including data from the media (both video and 
audio) itself. 

In he example illustrated in FIGS. 5 and 6. a color adjust 
60 effect has been selected and placed on an FX track 86. The 
selected clips which are positioned in video track VI during 
he tune sequence covered by he color adjust filter will be 
processed through that filter. Thus, a special effect may be 
associated with a particular sequence in a video track 
65 through time correlation of he effect with he dips in that 
tract By utilizing a separate FX track, the effect is assod- 
ated with a time sequence in he program rather than a 


35 
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parti cular dip or clips. Thus, for example, one or more of the 
dips may be removed and/or replaced with another clip by 
simply clicking and dragging the appropriate dip into posi- 
tion in the video track VI, and the preselected filter will act 
on whatever clip has been placed in the corresponding time 
sequence occupied by the filter in the FX track FX1. 

Referring to FIG. 6, the graphical display of the 
Sequencer Window 70 indicates the dataflow of the media 
for each point in time (i.e„ for cadi frame ) of th e program. 
Bv 'ccr rel thTa Tsel ected^poinf" 

in tin^(or^a*select^ u ser can glance along the 

time line cursor 106 in die video and FX tracks to ascertain 
what media is input through what effects filter or filters at 
each frame of the program. It should be noted that the system 
allows the user to add as many FX tracks as desired for any 
video track, so that each filter, as well as each media clip is 
independently displayed in vertically stacked tracks which 
are each synchronized with a horizontal time line. 


n^mayiaU^f^the 

creationrand editing, of multiple video tracks as* well as 

FIGF6r a^secondvideo 
track, V2, as well as other multiple tracks are combined 
during rendering by utilizing conventional compositing (or 
layering) techniques. In one embodiment, for example, the 
system performs an alpha composite of each of the synchro- 
nized multiple video tracks of a program on a frame-by- 
frame basis during rendering. Other composite techniques 
could, of course, be utilized without departing from the spirit 
of the present invention. 


It will be appreciated that the independent FX tracks is 
advantageous to the efficient operation of the system as well 
as being a useful graphical interface for the operator. By 
correlating the digitized media in with the one or more 
selected effects filters to a common parameter such as 
elapsed rime or frame number, a system can process any 
time-synchronized, or otherwise correlated media and 
filters, regardless of whether the dips providing the media 
input at any sdected point in time have been changed. Thus, 
in contrast to previous image processing regimes which 
associate a particular processing operation with particular 
selected media, the correlation of effects and media with a 
common parameter, such as time, allows for substitution of 
media or effects, with minimal restructuring of the process- 


ing. 

Referring to FIG. 5, in addition to creating and 
assembling reference clips into a program, the system of the 
present invention allows the user to implement a variety of 
transitions between the clips. The most common transition is 
a cut, which is an instantaneous switch from one dip to the 
next A cut transition occurs whenever two clips are arranged 
adjacently in the Sequencer Window. The system also pref- 
erably includes several conventional transitions such as a 
cross-fade audio transition, a dissolve transition, an iris 
transition, a radial wipe transition, and a wipe transition. Tb 
employ any of these transitions, the desired transition is 
selected from a sub- window which is opened by clicking on 
tiie app r op r i ate command from a conventional menu bar. 
The desired transition is then sdected and dragged from the 
pop-up window to the junction point between the two clips 
for which the transition is desired. In one embodiment, the 
system displays the transition as a two-toned gold strip 
overlaying the clip junction. 

In FIG. 5, a dissolve transition 84 has been placed 
between the “ledge shot” clip 90 and the adjacent “be on 
glader” clip 92. The duration of the transition, as well as any 
other parameters associated with the transition, may be set 
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by opening a transition dialog box using conventional menu 
activation techniques. In one embodiment of the present 
invention, the video track containing a transition may be 
expanded to graphically illustrate how the transition is 
3 positioned In relation to the dips it is used with. For 
example, as shown in FIG. 6, by activating toggle button 
108, video track VI is expanded vertically so that the “ledge 
shot” clip is shown positioned in a trade above the dissolve 
transition, which in turn is positioned above the “be on 
10 glacier” dip. Each of the dips, as well as the transition, 
remain correlated with the time line 72 in the Sequencer 
Window 70. This expanded view may illustrate, for 
example, that one of the dips does not extend through the 
entire period of the transition. Thus, “ledge shot” ends 
13 before completion of the dissolve transition. This is illus- 
trated by a shaded section 110, indicating an absence of 
media in the ‘ledge shot” clip for a portion of the dissolve 
transition. It will be appreciated that, since the system 
maintains the media tracks and transitions as independent 
2P entities, a desirable effect or transition, once created can be 
easily copied and moved to another location in the program. 
Thus, for example, the dissolve transition could be copied 
and moved to the junction between the “be on glacier" clip 
and the “twist shot" dip, without altering or affecting any of 
25 the media. 

The system preferably provides several ways to view a 
transition before rendering the entire program. By clicking 
on the transition 84 on the time line in the Sequencer 
Window 70 and dragging the time Line cursor 112 through 
30 the transition, a low quality, high-speed version of the 
transition is displayed in the FX monitor sub-window 120 
(shown in FIG. 9) of the Monitors Window 46. By clicking 
on the transition 84 in the Sequencer Window 70 and 
activating another pre-selected command, an animation of 
35 the transition can be viewed. Finally, by moving the time 
lin e: 106 to a selected frame in the transition 84 and acti- 
vating another preselected command, a snap shot of one 
frame of the program, at that point in the transition, is 
displayed at final image quality. It will be appreciated that 
40 these various preview options allow the user to select an 
option that hflianrra the conflicting considerations of mini- 
mizing processing time and maximizing image quality dur- 
ing the editing stage. 

Referring to FIG. 8. it will be appreciated by those skilled 
45 in the art of digital video image processing that the graphical 
display of the Sequencer Window 70 suggests a frame-by- 
frame dataflow far the program. For example, a transition 
(particularly when displayed in the expanded view shown in 
FIG. 7) graphically illustrates that at any point in time (i.e., 
50 at any single frame) there arc video inputs (Le., image 
frames) from two clips 90 and 92 which are processed as 
inputs through a transition filter 84 to yield a single output 
frame 95. And, in the example illustrated in FIGS. 5 and 6, 
at any selected paint in time during the transition 84, the 
55 output frame 95 is then processed by a special effects filter 
86 to yield a final frame in video track VI at the selected 
point in time. The Sequencer Window 70 thus illustrates the 
concept of program linearity by correlating the individual 
media and FX tracks with a time line, while suggesting the 
60 dataflow of any image processing performed on the media at 
any selected point in time merely by glancing vertically 
down the time cursor line 106 in the window. 

Referring to FIGS. 5 and 9, the system 20 may employ, 
and display, separate render tracks 82, 83 corresponding to 
65 the media tracks in a program. When displayed, the render 
track 82 for a corresponding video track 74 graphically 
indicates, as, for example, by shading, those portions of the 
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program which will require re-rendering, and those portions 
of the program which do not require re-rendering, since the 
last rendering. In the example shown in FIG. 5 , that portion 
85 of the render track 82 to the left of the time Hne cursor 
106 is shown by relatively lighter shading to have already 
been rendered, while the portion 87 of the render track 82 to 
the right of the time line cursor 106 is shown, by relatively 
darker shading, to require rendering. By keeping track of 
what portions of the program media require rendering since 
the last edit, the system 20 limits time con sumin g repro- 
cessing . 

FIG. 9 illustrates a simple algorithm for determining what 
portions of a program require rendering. In a routine that is 
called periodically, such as an interrupt routine, the system 
20 checks to determine whether any editing action has been 
taken by the operator. For example, the system checks, at 
202 to determine whether a dip has been inserted ca- 
re moved from a track. If. for example, a video clip has been 
inserted or removed, the system would then, at 204, adjust 
die downstream in and out paints on the media on the 
affected track to correlate the downstream in and out points 
on the render track with the downstream media on the media 
track that has been repositioned as a result of the insertion 
or deletion of the dip. For example, if the “mega j ump " Hip 
(listed in the Project Window 60 in FIG. 4) is inserted in 
video track 1 before the “twist shot” dip, the in and out 
points of the 'twist shot” dip and any downstream clips 
would be adjusted on both the video track 74 and the render 
track 82 by 19: 19. The system 20 would then mark the in 
point of the “mega jump” dip (in this example, at the timp 
line cursor 106 in the Sequencer Window 70 in FIG. 5) and 
the out point (19:19 frames later) of the render track to 
indicate that this segment must be re-rendered. 

The system 20 similarly checks, at 208, to see if a key 
frame has been added or removed. If so, some portion of the 
clip, including frames before and after the added or removed 
key frame, will be marked, at 206. on the render track, to 
again indicate that that portion of the media must be 
re-rendered. It should be noted that the location and number 
of frames affected by a change in a key frame will be 
dependent upon the type of inteipolation (i.e„ linear, 
curvilinear, or discontinuous) employed. 

The system 20 also checks, at 210, to determine whether 
any effects filters or transitions have been added or removed. 

If so, the system 20 marks the corresponding in and out 
points on the render track to indicate that re-rendering is 
required. 

It will be appreciated that the system may allow for, and 
must therefore monitor, other operations which might , 
modify the media on a particular track. Any such operations 
should be analyzed, the affected segments identified, and the 
appropriate in and out points marked for re-rendering on the 
render track. 

In one embodiment, the system 20 allows the operator to « 
manually mark portions of the render track 82 for 
re-rendering. Thus, for example, if a portion of segment 85 
of render track 82 were so marked, the potions marked for 
re -rendering would then be displayed in the relatively darks 
shading (as in segment 87). indicating that the corespond- < 
ing portions of the associated media tracks would be 
re-rendered on the next instruction to the system to render 
the program. 

Referring to FIG. 10. the Monitors Window 46 may also 
present an FX View in which an FX monitor sub-window 6 
120 is displayed adjacent the source monitor sub-window 
48. As with previewing transitions, the operator r»n preview 


12 

special effects created with filters in several ways. First, a 
high speed (i.e.. relatively rapidly processed) version of a 
filter operation may be viewed at relatively lower video 
quality by clicking the filter on the time line in the Sequencer 
5 Window 70 and dragging the time line cursor through the 
filter. The results will be displayed in the FX monitor 
sub- window 120 of the Monitors Window 46. Similarly, the 
fully processed effect of a filter at final image quality may be 
viewed by clicking on the filter in the time line in the 
10 Sequencer Window 70 and pressing a preselected command 
(such as the Shift key) while dragging the tin** line cursor 
through the filter. Again, the results are displayed in the FX 
monitor sub-window 120. Another method of previewing 
the effect of a particular filter, reviewing an animation of the 
15 filter, can be accomplished by selecting the filter on the time 
line in the Sequencer Window 70 and clicking the animate 
button (or selecting some other predefined keystroke). 
Reviewing an an i mati on of a filter's operation is particularly 
useful because it provides die operator with a view of how 
2 o the filter works in “real time” with minimal processing, since 
the actual video media is not processed at each frame. This 
preview thus provides the operator with an opportunity to 
adjust the pacing of the filter before any rendering of the 
program. 

25 Finally, a snap shot of any single selected frame of the 
program, including any special effects filters, may be viewed 
at final quality by moving the time line in die Sequencer 
Window 7# to the frame you want to see and clicking the 
snap shot button (or pressing some other preselected 
30 command). 

Referring to FIG. 11, the Monitors Window 46 may also 
provide a Trim View which includes two Trim monitor 
sub-windows 130 and 132 far displaying selected portions 
of adjacent clips during roll editing. Roll editing lets the 
35 operator edit adjacent clips in a trade — shortening one while 
simultaneously lengthening the other in order to maintain 
the constant length of the combined adjacent dips. The Trim 
View of the Monitors Window 46 may be activated by 
pressing a “Trim” button on the Monitors Window itself or 
40 by selecting the appropriate command from a pull-down 
menu, or by implementing a predefined keystroke on the 
keyboard. When activated, the selected adjacent clips are 
displayed in each of the Trim monitor sub-windows 130 and 
132, with the selected end frame of the first of the two 
« adjacent dips displayed in the left sub- window 130, and the 
selected first frame of the adjacent clips displayed in the 
right sub-window 132. By activating the appropriate buttons 
in the Monitors Window 46, the length of the first clip can 
be shortened and the second clip extended, or vice versa, 
so Then by clicking the play button, or by activating a similar 
preselected command at the keyboard or on the menu bar, 
the adjacent clips can be played to view the roll edited 
transition. 

Referring to FI G. 12, the system 20 of the present 
a invention may also provide a Picture View of the Project 
Window 60 wherein the clip sub-window 62 displays a 
selected portion, or all of the clips located in a selected bin 
in an adjustable array of frame windows in which an image 
of a selected frame, such as the first frame, of each of the 
0 clips in the bin is displayed. The name of the clip may also 
be displayed on a title bar 142 associated with each clip 
window 140. The size of the clip sub-window 62 can be 
varied, preferably through the use of conventional slider bars 
144 to size the clip sub-window 62 to display the desired 
5 number of dip windows 140 in a desired end-by-end array. 
The Picture View of the Project Window 60 may be utilized 
to order the dips in a desired sequence by clicking on and 


09/17/2004, EAST Version: 1.4.1 



5,682,326 


13 

dragging the dips to empty blocks in the Picture View 62, 
in order to achieve a preliminary sequence of display of the 
dips for a program. 

Referring to FIG. 13, the system 20 of the present 
invention may include a composite clip data hie structure for 5 
any rendered movies. The composite clip data file 150 
indudes a media segment 152 which contains the digital 
data corresponding to the rendered media in a conventional 
format In one embodiment, for example, the media segment 
152 of the composite dip data hie could be in Apple 1° 
QuickTime® format The composite clip data hie 150 also 
indudes a composition data segment 154 which indudes 
references to each of the source clips, references to any 
special effects and/or transitions, including established 
parameters and key frames, and the requisite timing inf or- 15 
mation required to reassemble the media contained in die 
media segment 152 of the hie. It will be appreciated that die 
existence of this composition information, properly stored 
directly with the media in a composite file structure, allows 
any systems that are capable of playing back the media to 20 
open the file and play the media, typically as a digital movie. 

For example, if the media portion of the file is in a Quick- 
Time® format, anyone with a copy of the composite clip 
data file can play the digital movie contained in the media 
file portion 152 of the data file if they have a Quicklime® 25 
system. In addition, if the composite clip data file 156 is read 
by a system, such as the present invention, capable of 
accessing and interpreting the composition data segment 154 
of the file, the system has access to the media required to 
play back the movie, as well as to all of the information that 30 
created the movie. This composite information thus allows 
the dip to be quickly edited and recreated by an appropriate 
system where the source material can be re-digitized or 
accessed from storage. 

While the best mode far carrying out die invention has 
been described in detail, those familiar wifi) the art to which 
this invention relates will recognize various alternative 
designs and embodiments for practicing the invention as 
disclosed by the following claims. 

What is claimed is: 40 

1. A digital movie processing system including a general 
purpose computer having: 
a processor, 

random access memory suitably connected to the proces- 45 
sot; 

storage memory operably connected to file processor, 
an input connected to the processor; and 
a display operably connected to the processor, the display 
including a plurality of windows, each of the windows 50 
having a different format, and wherein one of the 
windows provides a graphical time-line view of one or 
more clips of the movie, including at least one hod- 
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zontally extending media track in which graphical 
indicia of media dips can be inserted and correlated to 
an independent time line, and at least one special effects 
track in which graphical Indicia of a special effect can 
be inserted and correlated with the time line. 

2. A system fen processing digital movies including a 
general purpose computer having, 

a processor, 

random access memory operably connected to the 
processor, 

a display operably connected to the processor, 

an Input operably connected to file processor, and 

logic for correlating at least one media track containing at 
least one digital movie dip, and at least one FX track 
containing at least one special effect, with a common 
reference, thereby providing synchronization of the 
media track with the FX track to facilitate processing of 
the digital movie to obtain the special effect with the 
appropriate timing and parameters. 

3. In a system for processing digital movies including a 
computer having a processor, random access memory, stor- 
age memory, and logic for performing selected image pro- 
cessing operations on the digital movie, improved logic for 
rendering the movie, the improved logic including: 

(1) logic for identifying and marking digital information 
corresponding to those portions of the movie that have 
not been modified since the last time the movie was 
rendered, (2) logic for identifying and marking the 
digital data for those portions erf the movie that have 
been modified since the last time the movie was 
rendered, and (3) logic for identifying and marking the 
digital data for those portions of the movie that have not 
been modified since the last rendering but are affected 
by portions of the movie that have been modified since 
the last rendering, and logic for rendering only the 
digital data marked in sections (2) and (3) in response 
to an instruction to render the movie. 

4. In a system for processing digital movies including a 
computer having a processor, random access memory, stor- 
age memory, and logic for performing image processing 
operations on digital data corresponding to at least one 
movie clip, an improved composite clip data file comprising 
an image media segment and a composition data segment, 
the image media segment including digital data correspond- 
ing to the movie dip in a format suitable for playing the 
movie, and the composition data segment induding data 
corresponding to file identity and location of the source 
media and image processing operations performed on the 
source media, including processing and timing parameters, 
suitable for reproducing the movie clip contained in the 
media segment of the data file from the source dips. 

* * * * * 
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